In H.264/AVC (Advanced Video Coding), which is a video coding standard, the profiles of High Profile or higher can use a quantization step whose size differs from one component of an orthogonal transform coefficient to another for the quantization of image data. The quantization step for each component of the orthogonal transform coefficient may be set based on a reference step value and a quantization matrix (also referred to as a scaling list) defined by a size equivalent to the unit of orthogonal transform.
For example, a specified value of the quantization matrix is determined for each of a transform unit of a 4×4 size in the intra-prediction mode, a transform unit of a 4×4 size in the inter-prediction mode, a transform unit of an 8×8 size in the intra-prediction mode, and a transform unit of an 8×8 size in the inter-prediction mode. Further, users are allowed to specify a unique quantization matrix different from the specified values in a sequence parameter set or a picture parameter set. If no quantization matrices are used, the value of the quantization step to be used for quantization is equal for all the components.
In HEVC (High Efficiency Video Coding), which is under standardization as a next-generation video coding standard and which is a successor to H.264/AVC, the concept of CUs (Coding Units) corresponding to existing macroblocks has been introduced (see, for example, NPL 1). The range of coding unit sizes is specified by a set of values which are powers of 2, called LCU (Largest Coding Unit) and SCU (Smallest Coding Unit), in a sequence parameter set. A specific coding unit size in the range specified by the LCU and the SCU is specified using split_flag.
In HEVC, one coding unit can be partitioned into one or more orthogonal transform units, or one or more transform units (TUs). Any of 4×4, 8×8, 16×16, and 32×32 is available as the size of a transform unit. Accordingly, a quantization matrix can also be specified for each of those candidate transform unit sizes.
In H.264/AVC, only one quantization matrix can be specified in each picture for the size of each transform unit. In contrast, it has been proposed to specify multiple candidate quantization matrices in each picture for the size of each transform unit and adaptively select a quantization matrix for each block in terms of RD (Rate-Distortion) optimization (see, for example, NPL 2).